The present invention relates in general to enterprise information systems. More particularly, the present invention relates to methods and apparatus for optimizing the delivery of data to a device.
Timely intelligence is critical to proper business decision making. Collecting and quickly analyzing information, however, is often difficult. In a dynamic marketplace, delayed delivery of intelligence can reduce both its reliability and relevancy. Substantial delays can even make the information, often acquired at considerable cost, completely worthless.
Intelligence is often extracted from OLTP applications and from specialized databases, called operational data stores, OLTP (online transaction processing) applications are those enterprise systems that manage a company's basic transactions, such as order entry and billing systems. Since they are commonly optimized for reading and writing, and not for querying, extracting information from an OLTP application can be sufficiently complex to require additional user training. Furthermore, while operational data stores generally archive OLTP information in a format for querying, they also generally do not maintain important historical information. For instance, an operational data store may store a current checking account balance, but not the individual daily balances over the previous month.
In addition, the queries themselves often take substantial time to execute, producing only static snapshots of the information. Observing the dynamic aspect of changing information is therefore difficult. A possible solution may be to sequentially execute and aggregate a series of queries. However, this solution can be both inefficient and ineffective, since manual manipulation still often delays the information delivery. That is, these queries must be first manually aggregated and summarized, before intelligence reports can be generated and delivered.
ETL (extraction, transformation, and loading) systems help by extracting, transforming, and aggregating the information. But latency inherently found in most distributed networks, coupled with considerable manual intervention that ETL systems often require, mean that critical information can still be received late.
Automatic notification is a possible solution. Many analytical products such as business intelligence (BI) and online analytical processing (OLAP) systems are capable of monitoring, scheduling, and broadcasting alerts via email or pager. These systems, however, cannot generally assure that the intended recipient promptly receives and reads the message. This presents a significant problem for time-sensitive information, where minutes or even seconds can make a difference.
To facilitate discussion, FIG. 1 shows a simplified functional diagram of distributed information architecture. The diagram can be divided into an enterprise data layer 160 and a client layer 162. Enterprise data layer 160 comprises elements that are primarily focused on accumulating, processing, and transforming operational data. Client layer 162 comprises elements that are primarily focused on rendering the processed data for a user.
OLTP (online transaction processing) applications 152 are commonly coupled to each other, as well as to other enterprise applications, through a dedicated messaging and queuing application (MQ), such as IBM's MQSeries. MQ provides an efficient communication channel for these applications, by storing and forwarding data messages, in a manner that is similar to email.
Commonly coupled to each OLTP application 152 is operational data store 154, such as an Oracle database. Through an API (application programming interface), transactional data can be transferred between the OLTP application and the database. Operational data store 154 consolidates that data from multiple sources and provides a near real-time, integrated view of volatile, current data. Since its purpose is to provide integrated data for operational purposes, operational data store 154 primarily has add, change, and delete functionality.
In order to conduct meaningful analysis, this information is often further placed in a more stable environment, optimized for random querying. ETL system 155 extracts the information from the appropriate data store 154, transforms and combines the data based on pre-defined constraints, and subsequently loads the data into data warehouse 156. A popular ETL technique, developed by Sagent, is the use of data flows.
Data flows are a series of rule-enabled transformations that are connected in data pipelines. They handle the tasks of joining, merging, comparing and splitting data and permit the separation of data into different logic paths, each of which can be further combined and split off to create more complex transformation sequences.
ETL data extractions often occur by either a bulk or a trickle method. In the bulk method, periodic snap shots of data in operational data store 154 are extracted and uploaded into data warehouse 156. This commonly occurs as a large batch file scheduled during a low system utilization period. In the trickle method, changes in operational data store 154 are continuously uploaded, or “trickled” into data warehouse 156. These updates are therefore frequent, smaller, and more current than in the bulk method. As in the case of OLTP 152 systems, ETL 155 can also use the MQ for data extraction.
Once the data is in data warehouse 156, it is available for OLAP 158 (online analytical processing). OLAP enables trained users to perform ad hoc analysis of data in multiple dimensions, such as with an OLAP cube. OLAP cubes provide multi-dimensional views of data, querying, and analytical capabilities. Furthermore, many OLAP products can schedule, run, publish, and broadcast reports, alerts and responses over the network, email, or personal digital assistant. Users often access OLAP 158 by thin client 162. Thin clients are applications that generally are integrated into the underlying client device, and generally require minimal modification. For instance, a thin client can be browser with a Macromedia Flash module installed.
Although OLAP analysis can provide valuable insight about business operations, critical information is often received late, even with automated reporting. Automated OLAP reporting often only has access to the information within data warehouse 156, which can be several processing stages behind OLTP 152. This delay can be substantial, reducing the information's value. Furthermore, these reports are often only static snapshots of information in data warehouse 156.
For example, a NASDAQ broker places an order into an OLTP 152 application called an electronic communications network, or ECN. The ECN matches customer buy and sell orders directly through the computer. In this case, an order to buy 100 shares of ABC at $18.75 was entered. This open order is stored in the ECN operational data store 154, subsequently extracted by ETL 155, and analyzed by OLAP 158. If the buy order amount is the then highest in the ECN, OLAP 158 forwards the information to thin client 164, NASDAQ quote montage, where it is immediately displayed on the familiar stock market ticker tape. And although this system delivers stock information to individual brokers with reasonably small latency, it is also not easily modified. The NASDAQ application is custom designed for the specific purpose of enabling stock trading. As such, it would be difficult to display additional data on the stock ticker, such as non-financial information, without substantial additional programming.
In view of the foregoing, there is desired a method and apparatus for optimizing the delivery of data to a device, in which relevant information is received in a timely manner, and in which that data is rendered in a dynamic format.